{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyHealth Datasets Overview\n",
    "\n",
    "This notebook provides an overview of all datasets supported in PyHealth. For each dataset, we include:\n",
    "- **Description**: What the dataset contains.\n",
    "- **Source/Link**: Where to find the original data.\n",
    "- **Download Method**: How to obtain the data.\n",
    "- **Restrictions**: Any access requirements or limitations.\n",
    "- **Example Usage**: Code to load the dataset in PyHealth.\n",
    "\n",
    "**Note**: Many datasets require accounts, credentials, or compliance with data use agreements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. MIMIC-III\n",
    "\n",
    "**Description**: A large dataset of de-identified health records from ICU patients at Beth Israel Deaconess Medical Center (2001-2012). Includes demographics, vital signs, lab results, medications, and more.\n",
    "\n",
    "**Source/Link**: https://physionet.org/content/mimiciii/1.4/\n",
    "\n",
    "**Download Method**:\n",
    "- Create PhysioNet account and complete HIPAA training.\n",
    "- Download: `wget -r -N -c -np --user [USERNAME] --ask-password https://physionet.org/files/mimiciii/1.4/`\n",
    "- Demo (no auth): `wget -r -N -c -np https://physionet.org/files/mimiciii-demo/1.4/`\n",
    "\n",
    "**Restrictions**: Requires PhysioNet account, HIPAA certification, and data use agreement. ~40GB.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No config path provided, using default config\n",
      "Initializing mimic3 dataset from https://physionet.org/files/mimiciii-demo/1.4/ (dev mode: True)\n",
      "Scanning table: patients from https://physionet.org/files/mimiciii-demo/1.4/PATIENTS.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/PATIENTS.csv\n",
      "Scanning table: admissions from https://physionet.org/files/mimiciii-demo/1.4/ADMISSIONS.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/ADMISSIONS.csv\n",
      "Scanning table: icustays from https://physionet.org/files/mimiciii-demo/1.4/ICUSTAYS.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/ICUSTAYS.csv\n",
      "Scanning table: diagnoses_icd from https://physionet.org/files/mimiciii-demo/1.4/DIAGNOSES_ICD.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/DIAGNOSES_ICD.csv\n",
      "Joining with table: https://physionet.org/files/mimiciii-demo/1.4/ADMISSIONS.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/ADMISSIONS.csv\n",
      "Scanning table: prescriptions from https://physionet.org/files/mimiciii-demo/1.4/PRESCRIPTIONS.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/PRESCRIPTIONS.csv\n",
      "Joining with table: https://physionet.org/files/mimiciii-demo/1.4/ADMISSIONS.csv.gz\n",
      "Original path does not exist. Using alternative: https://physionet.org/files/mimiciii-demo/1.4/ADMISSIONS.csv\n"
     ]
    }
   ],
   "source": [
    "from pyhealth.datasets import MIMIC3Dataset\n",
    "\n",
    "# Download Demo MIMIC3 dataset\n",
    "mimic3_demo = MIMIC3Dataset(\n",
    "    root=\"https://physionet.org/files/mimiciii-demo/1.4/\",\n",
    "    tables=[\"DIAGNOSES_ICD\", \"PRESCRIPTIONS\"],\n",
    "    dev=True  # Use dev mode for small subset\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting global event dataframe...\n",
      "Dev mode enabled: limiting to 1000 patients\n",
      "Collected dataframe with shape: (12524, 46)\n",
      "Dataset: mimic3\n",
      "Dev mode: True\n",
      "Number of patients: 100\n",
      "Number of events: 12524\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(mimic3_demo.stats())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. MIMIC-IV\n",
    "\n",
    "**Description**: Updated version of MIMIC-III with EHR, clinical notes, and chest X-rays from 2008-2019.\n",
    "\n",
    "**Source/Link**: https://physionet.org/content/mimiciv/0.4/\n",
    "\n",
    "**Download Method**:\n",
    "- PhysioNet account required.\n",
    "- Download: `wget -r -N -c -np --user [USERNAME] --ask-password https://physionet.org/files/mimiciv/2.2/`\n",
    "- Demo: `wget -r -N -c -np https://physionet.org/files/mimic-iv-demo/2.2/`\n",
    "\n",
    "**Restrictions**: PhysioNet account, HIPAA training. ~200GB.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage Starting MIMIC4Dataset init: 806.7 MB\n",
      "Initializing MIMIC4EHRDataset with tables: ['diagnoses_icd', 'prescriptions'] (dev mode: True)\n",
      "Using default EHR config: /home/ubuntu/PyHealth/pyhealth/datasets/configs/mimic4_ehr.yaml\n",
      "Memory usage Before initializing mimic4_ehr: 806.7 MB\n",
      "Initializing mimic4_ehr dataset from https://physionet.org/files/mimic-iv-demo/2.2/ (dev mode: False)\n",
      "Scanning table: diagnoses_icd from https://physionet.org/files/mimic-iv-demo/2.2/hosp/diagnoses_icd.csv.gz\n",
      "Joining with table: https://physionet.org/files/mimic-iv-demo/2.2/hosp/admissions.csv.gz\n",
      "Scanning table: prescriptions from https://physionet.org/files/mimic-iv-demo/2.2/hosp/prescriptions.csv.gz\n",
      "Scanning table: patients from https://physionet.org/files/mimic-iv-demo/2.2/hosp/patients.csv.gz\n",
      "Scanning table: admissions from https://physionet.org/files/mimic-iv-demo/2.2/hosp/admissions.csv.gz\n",
      "Scanning table: icustays from https://physionet.org/files/mimic-iv-demo/2.2/icu/icustays.csv.gz\n",
      "Memory usage After initializing mimic4_ehr: 806.7 MB\n",
      "Memory usage After EHR dataset initialization: 806.7 MB\n",
      "Memory usage Before combining data: 806.7 MB\n",
      "Combining data from ehr dataset\n",
      "Creating combined dataframe\n",
      "Memory usage After combining data: 806.7 MB\n",
      "Memory usage Completed MIMIC4Dataset init: 806.7 MB\n"
     ]
    }
   ],
   "source": [
    "from pyhealth.datasets import MIMIC4Dataset\n",
    "\n",
    "# EHR only\n",
    "mimic4_demo = MIMIC4Dataset(\n",
    "    ehr_root=\"https://physionet.org/files/mimic-iv-demo/2.2/\",\n",
    "    ehr_tables=[\"diagnoses_icd\", \"prescriptions\"],\n",
    "    dev=True\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting global event dataframe...\n",
      "Dev mode enabled: limiting to 1000 patients\n",
      "Collected dataframe with shape: (23108, 34)\n",
      "Dataset: mimic4\n",
      "Dev mode: True\n",
      "Number of patients: 100\n",
      "Number of events: 23108\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(mimic4_demo.stats())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. eICU\n",
    "\n",
    "**Description**: Multi-center ICU database from US hospitals (2014-2015), including demographics, diagnoses, treatments, labs, and vitals.\n",
    "\n",
    "**Source/Link**: https://eicu-crd.mit.edu/\n",
    "\n",
    "**Download Method**:\n",
    "- Register and agree to terms.\n",
    "- Download: `wget -r -N -c -np --user [USERNAME] --ask-password https://physionet.org/files/eicu-crd/2.0/`\n",
    "\n",
    "**Restrictions**: Account required, data use agreement. ~10GB.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyhealth.datasets import eICUDataset\n",
    "\n",
    "# No demo is available for EICU, so this will be empty unless you have the data locally\n",
    "eicu = eICUDataset(\n",
    "    root=\"/path/to/eicu-crd/2.0\",\n",
    "    tables=[\"diagnosis\", \"medication\"],\n",
    "    dev=True\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyhealth.datasets.eicu.eICUDataset at 0x76e5605c3650>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eicu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Datasets in OMOP format\n",
    "\n",
    "**Description**: Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM) is a standard for structuring healthcare data. PyHealth supports datasets formatted in OMOP-CDM (e.g., from various sources like EHR systems).\n",
    "\n",
    "**Source/Link**: https://www.ohdsi.org/data-standardization/the-common-data-model/\n",
    "\n",
    "**Download Method**: Varies by source; often requires partnerships or custom access to OMOP-formatted data.\n",
    "\n",
    "**Restrictions**: Institutional access often required.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from pyhealth.datasets import OMOPDataset\n",
    "\n",
    "dataset = OMOPDataset(\n",
    "    root=\"/path/to/omop/data\",\n",
    "    tables=[\"condition_occurrence\", \"drug_exposure\"]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Sleep-EDF\n",
    "\n",
    "**Description**: Sleep EEG recordings for sleep staging.\n",
    "\n",
    "**Source/Link**: https://physionet.org/content/sleep-edfx/1.0.0/\n",
    "\n",
    "**Download Method**: `wget -r -N -c -np https://physionet.org/files/sleep-edfx/1.0.0/`\n",
    "\n",
    "**Restrictions**: Publicly available, no auth.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from pyhealth.datasets import SleepEDFDataset\n",
    "\n",
    "dataset = SleepEDFDataset(\n",
    "    root=\"/path/to/sleep-edfx\",\n",
    "    dev=True\n",
    ")\n",
    "print(dataset.stats())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. SHHS\n",
    "\n",
    "**Description**: Sleep Heart Health Study polysomnography data.\n",
    "\n",
    "**Source/Link**: https://sleepdata.org/datasets/shhs\n",
    "\n",
    "**Download Method**: Register and download from site.\n",
    "\n",
    "**Restrictions**: Account required.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from pyhealth.datasets import SHHSDataset\n",
    "\n",
    "dataset = SHHSDataset(\n",
    "    root=\"/path/to/shhs\",\n",
    "    dev=True\n",
    ")\n",
    "print(f\"Loaded {len(dataset.patients)} patients.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. ISRUC\n",
    "\n",
    "**Description**: ISRUC-SLEEP dataset for sleep staging.\n",
    "\n",
    "**Source/Link**: https://sleeptight.isr.uc.pt/?page_id=48\n",
    "\n",
    "**Download Method**: Download from site.\n",
    "\n",
    "**Restrictions**: Public.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from pyhealth.datasets import ISRUCDataset\n",
    "\n",
    "dataset = ISRUCDataset(\n",
    "    root=\"/path/to/isruc\",\n",
    "    dev=True\n",
    ")\n",
    "print(f\"Loaded {len(dataset.patients)} patients.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Cardiology (PhysioNet Challenge 2020)\n",
    "\n",
    "**Description**: ECG data from multiple sources for arrhythmia detection.\n",
    "\n",
    "**Source/Link**: https://physionet.org/content/challenge-2020/1.0.2/\n",
    "\n",
    "**Download Method**: `wget -r -N -c -np https://physionet.org/files/challenge-2020/1.0.2/`\n",
    "\n",
    "**Restrictions**: Public.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<pyhealth.datasets.cardiology.CardiologyDataset object at 0x76e557bf53d0>\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from pyhealth.datasets import CardiologyDataset\n",
    "\n",
    "cardiology = CardiologyDataset(\n",
    "    #root=\"/path/to/challenge-2020\",\n",
    "    root = \"https://physionet.org/files/challenge-2020/1.0.2/\",\n",
    "    dev=True\n",
    ")\n",
    "print(cardiology)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. COVID-19 CXR\n",
    "\n",
    "**Description**: Chest X-ray images for COVID-19 classification.\n",
    "\n",
    "**Source/Link**: Custom or public sources (check PyHealth docs).\n",
    "\n",
    "**Download Method**: Varies; often from Kaggle or GitHub.\n",
    "\n",
    "**Restrictions**: Public datasets.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from pyhealth.datasets import COVID19CXRDataset\n",
    "\n",
    "dataset = COVID19CXRDataset(\n",
    "    root=\"/path/to/covid19-cxr\",\n",
    "    dev=True\n",
    ")\n",
    "print(f\"Loaded {len(dataset.patients)} patients.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Sample Dataset (Test/Synthetic)\n",
    "\n",
    "**Description**: Synthetic dataset for testing and development, with customizable samples.\n",
    "\n",
    "**Source/Link**: Built-in to PyHealth (no external source).\n",
    "\n",
    "**Download Method**: No download needed; created programmatically.\n",
    "\n",
    "**Restrictions**: None; for testing only.\n",
    "\n",
    "**Example Usage**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mInit signature:\u001b[39m\n",
      "SampleDataset(\n",
      "    samples: List[Dict],\n",
      "    input_schema: Dict[str, Union[str, Type[pyhealth.processors.base_processor.FeatureProcessor]]],\n",
      "    output_schema: Dict[str, Union[str, Type[pyhealth.processors.base_processor.FeatureProcessor]]],\n",
      "    dataset_name: Optional[str] = \u001b[38;5;28;01mNone\u001b[39;00m,\n",
      "    task_name: Optional[str] = \u001b[38;5;28;01mNone\u001b[39;00m,\n",
      "    input_processors: Optional[Dict[str, pyhealth.processors.base_processor.FeatureProcessor]] = \u001b[38;5;28;01mNone\u001b[39;00m,\n",
      "    output_processors: Optional[Dict[str, pyhealth.processors.base_processor.FeatureProcessor]] = \u001b[38;5;28;01mNone\u001b[39;00m,\n",
      ") -> \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "\u001b[31mDocstring:\u001b[39m     \n",
      "Sample dataset class for handling and processing data samples.\n",
      "\n",
      "Attributes:\n",
      "    samples (List[Dict]): List of data samples.\n",
      "    input_schema (Dict[str, Union[str, Type[FeatureProcessor], Tuple[Union[str, Type[FeatureProcessor]], Dict[str, Any]]]]):\n",
      "        Schema for input data. Values can be string aliases, processor classes, or tuples of (spec, kwargs_dict).\n",
      "    output_schema (Dict[str, Union[str, Type[FeatureProcessor], Tuple[Union[str, Type[FeatureProcessor]], Dict[str, Any]]]]):\n",
      "        Schema for output data. Values can be string aliases, processor classes, or tuples of (spec, kwargs_dict).\n",
      "    dataset_name (Optional[str]): Name of the dataset.\n",
      "    task_name (Optional[str]): Name of the task.\n",
      "\u001b[31mInit docstring:\u001b[39m\n",
      "Initializes the SampleDataset with samples and schemas.\n",
      "\n",
      "Args:\n",
      "    samples (List[Dict]): List of data samples.\n",
      "    input_schema (Dict[str, Union[str, Type[FeatureProcessor], Tuple[Union[str, Type[FeatureProcessor]], Dict[str, Any]]]]):\n",
      "        Schema for input data. Values can be string aliases, processor classes, or tuples of (spec, kwargs_dict) for instantiation.\n",
      "    output_schema (Dict[str, Union[str, Type[FeatureProcessor], Tuple[Union[str, Type[FeatureProcessor]], Dict[str, Any]]]]):\n",
      "        Schema for output data. Values can be string aliases, processor classes, or tuples of (spec, kwargs_dict) for instantiation.\n",
      "    dataset_name (Optional[str], optional): Name of the dataset.\n",
      "        Defaults to None.\n",
      "    task_name (Optional[str], optional): Name of the task.\n",
      "        Defaults to None.\n",
      "    input_processors (Optional[Dict[str, FeatureProcessor]],\n",
      "        optional): Pre-fitted input processors. If provided, these\n",
      "        will be used instead of creating new ones from input_schema.\n",
      "        Defaults to None.\n",
      "    output_processors (Optional[Dict[str, FeatureProcessor]],\n",
      "        optional): Pre-fitted output processors. If provided, these\n",
      "        will be used instead of creating new ones from output_schema.\n",
      "        Defaults to None.\n",
      "\u001b[31mFile:\u001b[39m           ~/PyHealth/pyhealth/datasets/sample_dataset.py\n",
      "\u001b[31mType:\u001b[39m           type\n",
      "\u001b[31mSubclasses:\u001b[39m     "
     ]
    }
   ],
   "source": [
    "SampleDataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label label vocab: {0: 0, 1: 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processing samples:   0%|          | 0/2 [00:00<?, ?it/s]\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "unhashable type: 'list'",
     "output_type": "error",
     "traceback": [
      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
      "\u001b[31mTypeError\u001b[39m                                 Traceback (most recent call last)",
      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[60]\u001b[39m\u001b[32m, line 9\u001b[39m\n\u001b[32m      3\u001b[39m \u001b[38;5;66;03m# Create synthetic samples\u001b[39;00m\n\u001b[32m      4\u001b[39m samples = [\n\u001b[32m      5\u001b[39m     {\u001b[33m\"\u001b[39m\u001b[33mpatient_id\u001b[39m\u001b[33m\"\u001b[39m: \u001b[33m\"\u001b[39m\u001b[33m1\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mconditions\u001b[39m\u001b[33m\"\u001b[39m: [[\u001b[33m\"\u001b[39m\u001b[33mC001\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mC002\u001b[39m\u001b[33m\"\u001b[39m]], \u001b[33m\"\u001b[39m\u001b[33mlabel\u001b[39m\u001b[33m\"\u001b[39m: \u001b[32m0\u001b[39m},\n\u001b[32m      6\u001b[39m     {\u001b[33m\"\u001b[39m\u001b[33mpatient_id\u001b[39m\u001b[33m\"\u001b[39m: \u001b[33m\"\u001b[39m\u001b[33m2\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mconditions\u001b[39m\u001b[33m\"\u001b[39m: [[\u001b[33m\"\u001b[39m\u001b[33mC003\u001b[39m\u001b[33m\"\u001b[39m]], \u001b[33m\"\u001b[39m\u001b[33mlabel\u001b[39m\u001b[33m\"\u001b[39m: \u001b[32m1\u001b[39m}\n\u001b[32m      7\u001b[39m ]\n\u001b[32m----> \u001b[39m\u001b[32m9\u001b[39m dataset = \u001b[43mSampleDataset\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m     10\u001b[39m \u001b[43m    \u001b[49m\u001b[43msamples\u001b[49m\u001b[43m=\u001b[49m\u001b[43msamples\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m     11\u001b[39m \u001b[43m    \u001b[49m\u001b[43minput_schema\u001b[49m\u001b[43m=\u001b[49m\u001b[43m{\u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mconditions\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43msequence\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m}\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m     12\u001b[39m \u001b[43m    \u001b[49m\u001b[43moutput_schema\u001b[49m\u001b[43m=\u001b[49m\u001b[43m{\u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mlabel\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mbinary\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m}\u001b[49m\n\u001b[32m     13\u001b[39m \u001b[43m)\u001b[49m\n\u001b[32m     14\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mLoaded \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(dataset)\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m synthetic samples.\u001b[39m\u001b[33m\"\u001b[39m)\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/PyHealth/pyhealth/datasets/sample_dataset.py:88\u001b[39m, in \u001b[36mSampleDataset.__init__\u001b[39m\u001b[34m(self, samples, input_schema, output_schema, dataset_name, task_name, input_processors, output_processors)\u001b[39m\n\u001b[32m     85\u001b[39m         \u001b[38;5;28mself\u001b[39m.record_to_index[record_id].append(i)\n\u001b[32m     87\u001b[39m \u001b[38;5;28mself\u001b[39m.validate()\n\u001b[32m---> \u001b[39m\u001b[32m88\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mbuild\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/PyHealth/pyhealth/datasets/sample_dataset.py:150\u001b[39m, in \u001b[36mSampleDataset.build\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m    148\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m sample.items():\n\u001b[32m    149\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m k \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.input_processors:\n\u001b[32m--> \u001b[39m\u001b[32m150\u001b[39m         sample[k] = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43minput_processors\u001b[49m\u001b[43m[\u001b[49m\u001b[43mk\u001b[49m\u001b[43m]\u001b[49m\u001b[43m.\u001b[49m\u001b[43mprocess\u001b[49m\u001b[43m(\u001b[49m\u001b[43mv\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    151\u001b[39m     \u001b[38;5;28;01melif\u001b[39;00m k \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.output_processors:\n\u001b[32m    152\u001b[39m         sample[k] = \u001b[38;5;28mself\u001b[39m.output_processors[k].process(v)\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/PyHealth/pyhealth/processors/sequence_processor.py:38\u001b[39m, in \u001b[36mSequenceProcessor.process\u001b[39m\u001b[34m(self, value)\u001b[39m\n\u001b[32m     36\u001b[39m     indices.append(\u001b[38;5;28mself\u001b[39m.code_vocab[\u001b[33m\"\u001b[39m\u001b[33m<unk>\u001b[39m\u001b[33m\"\u001b[39m])\n\u001b[32m     37\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m---> \u001b[39m\u001b[32m38\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[43mtoken\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mnot\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mcode_vocab\u001b[49m:\n\u001b[32m     39\u001b[39m         \u001b[38;5;28mself\u001b[39m.code_vocab[token] = \u001b[38;5;28mself\u001b[39m._next_index\n\u001b[32m     40\u001b[39m         \u001b[38;5;28mself\u001b[39m._next_index += \u001b[32m1\u001b[39m\n",
      "\u001b[31mTypeError\u001b[39m: unhashable type: 'list'"
     ]
    }
   ],
   "source": [
    "from pyhealth.datasets import SampleDataset\n",
    "\n",
    "# Create synthetic samples\n",
    "samples = [\n",
    "    {\"patient_id\": \"1\", \"conditions\": [\"C001\", \"C002\"], \"label\": 0},\n",
    "    {\"patient_id\": \"2\", \"conditions\": [\"C003\"], \"label\": 1}\n",
    "]\n",
    "\n",
    "dataset = SampleDataset(\n",
    "    samples=samples,\n",
    "    input_schema={\"conditions\": \"sequence\"},\n",
    "    output_schema={\"label\": \"binary\"}\n",
    ")\n",
    "print(f\"Loaded {len(dataset)} synthetic samples.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Datasets\n",
    "\n",
    "- **DREAMT**: https://physionet.org/content/dreamt/ - Download from PhysioNet.\n",
    "- **EHRShot**: Benchmark dataset for EHR tasks.\n",
    "- **Medical Transcriptions**: Text data for NLP.\n",
    "- **BMD_HS**: Bone mineral density data.\n",
    "- **TUAB/TUEV**: EEG datasets from Temple University.\n",
    "- **MIMIC-Extract**: Processed MIMIC data.\n",
    "\n",
    "For full details, check the PyHealth documentation and dataset source links."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
